Apparatus and method for image display

ABSTRACT

A mobile data apparatus equipped with a display is improved in both operability and mobility.  
     A frame difference detector  4  is provided for detecting a change in image between two consecutive frames transferred from a pair of frame buffers  2  and  3.  The change in image is used in a barycenter calculator  5  for calculating the barycenter of each image. The barycenter is saved in a barycenter coordinates memory  6.  A camera movement calculator  6  calculates a difference between the barycenter saved in the barycenter coordinates memory  6  and a reference barycenter saved in a barycenter coordinates memory  7  to determine a horizontal pixel shift DPx and a vertical pixel shift DPy of a camera. In response to those shifts, a horizontal angle-of-rotation controller  9  and a vertical angle-of-rotation controller  10  determine angles of rotation along the horizontal and vertical directions hx and hy for a virtual camera  11.  Accordingly, the view of the virtual camera  11  can duly be controlled.

BACKGROUND OF THE INVENTION

[0001] 1. Field of the Invention

[0002] The present invention relates to an apparatus and a method fordisplaying an image of view point information produced with aremote-controlled camera.

[0003] 2. Description of Related Art

[0004] As a wide range of Internet services have overwhelmingly beenprovided, their displaying data include not only texts, still images,and sounds but also motion images and 3-D dynamic model spaces. Fordeveloping such 3-D dynamic model spaces, the VRML (virtual realitymodeling language) is standardized in August, 1996 and used commonly asa today's standard technology. This technology allows a variety of 3-Dmodel spaces of one format, which may simulate museums an theaters forexample, to be constructed as information to the Internet and accessedby any user who operates its computer connected to the Internet to enjoyas if he or she walks around in a museum or a theater in virtual sense.For the purpose, an image displayed on a display unit of the computer isbased on the view of a virtual camera installed virtually in the 3-Dmodel spaces. Upon receiving a command from the user requesting forwalking about in a 3-D model space, the computer controls the virtualcamera for desired positioning and orientation. As the result, an imageis displayed on the display of the computer showing as if the user walksabout in the 3-D model space.

[0005] The command to the computer may be given by the user operatingwith its hand or fingers a pointing device, such as a mouse or ajoystick, or a touch display in all upward, downward, leftward, andrightward directions.

[0006] The commanding of the computer with the use of a hand or fingersoperating the pointing device, such as a mouse or a joystick, or thetouch display in all upward, downward, leftward, and rightwarddirections is substantially suited for providing a modification or acommand on the screen displaying a relatively less motion image such asa text or a still image. It is however unfavorable to control quickmovements of a view point or the view of the virtual camera in 3-D modespaces. It is routine that when a person changes its view point in dailylife, its eyes or head is shifted to a desired location but rarely movesand orients the object to be viewed.

[0007] This drawback may not be critical with a desk-top type of thecomputer where the view of the user is fixed to its display screen at astationary location while controlling an indicator or pointer on thescreen by manipulating the control with the hand or fingers.

[0008] However, when a specific type or namely, a head-mounting type ofthe display is used, it is hung on the front of the eyes of the user andmoves together with the head. When a hand-held type of the display isused, it is held by the hand of the user and turned to any of upward,downward, leftward, and rightward directions in response to the movementof the head so that it can stay at the front of the user's eyes.

[0009] In a system equipped with a mobile type of the display which canbe held in the front of the eyes of a user, it is disadvantageous fordynamically shifting through a computer the view point of a virtualcamera in a 3-D model space to use a pointing device, such as a mouse ora joystick, or a touch device which can be controlled by the hand orfingers in upward, downward, leftward, and rightward directions.

[0010] It is also known in a virtual reality technology or atele-existence technology for controlling the dynamic movement of a viewpoint with a computer that the view point of a user is normally measuredby a measuring device, such as an eye-tracking device or a heat-positiontracing device, which can be fitted to a body of the user. The fittingof the device to the body will add another inconvenience to the user,impairing the operability and the mobility.

SUMMARY OF THE INVENTION

[0011] It is a primary object of the present invention to provide anapparatus and a method for displaying an image where the view point of avirtual camera can dynamically be shifted in response to the movement ofa mobile set equipped with a display.

[0012] It is another object of the present invention to provide anapparatus and a method for displaying an image where its mobile setequipped with a display is improved in both the operability and themobility.

[0013] An image displaying apparatus according to the present inventionincludes a controller which processes input image data produced by animage input unit which pictures the background of an image display unitin order to detect a movement of the image display unit, and an imagedata generator which generates data indicative of the view of a cameraremote controlled in response to the detected movement, whereby theimage data can be displayed on the image display unit.

[0014] An image displaying method according to the present inventioncomprises the steps of picturing the background of an image displayunit, processing pictured data to detect a movement of the image displayunit, generating image data of the view of a camera remote controlled inresponse to the detected movement, and displaying it on the imagedisplay unit.

BRIEF DESCRIPTION OF THE DRAWINGS

[0015]FIG. 1 is a block diagram of a functional arrangement of an imagedisplaying apparatus according to the present invention;

[0016]FIG. 2 is a flowchart showing actions of the image displayingapparatus;

[0017]FIG. 3 is a front view of a mobile data assistant associated withthe present invention;

[0018]FIG. 4 is a side view of the mobile data assistant.

DESCRIPTION OF PREFERRED EMBODIMENTS

[0019] Preferred embodiments of the present invention will be describedin more detail referring to the accompanying drawings.

[0020] An image displaying apparatus according to the present inventionhas an arrangement shown in FIG. 1. The image displaying apparatus is amobile type image displaying apparatus comprising a microcomputer 14,and an image input unit 1 and an image display unit 13 both connected tothe microcomputer 14.

[0021] The image input unit 1 comprises a CCD camera for picturing thebackground of the image display unit 13. The image display unit 13 is aliquid crystal display for displaying an image of visual informationsupplied from the microcomputer 14.

[0022] The microcomputer 14 includes, as best shown in the functionalblock diagram of FIG. 1, a frame difference detector 4 to which imagedata generated by the image input unit 1 is supplied via frame buffers 2and 3, a barycenter calculator 5 connected to the frame differencedetector 4, a camera movement calculator 8 connected via barycentercoordinates memories 6 and 7 to the barycenter calculator 5, a pair of ahorizontal angle-of-rotation controller 9 and a verticalangle-of-rotation controller 10 both connected to the camera movementcalculator 8, a virtual camera 11 connected to the horizontal andvertical angle-of-rotation controllers 9, 10, and a 3-D object spacedatabase 12 connected to the virtual camera 11 for constructing a 3-Dmodel space.

[0023] The image displaying apparatus performs a succession of actions,as shown in a flowchart of FIG. 2, including moving its body equippedwith the image display unit 13 (Step S1), saving in the frame buffer 2each frame of input image data generated by the image input unit 1imaging the background of the input display unit 13, and transferringthe saved data of the frame buffer 2 to the frame buffer 3 before thesucceeding frame is fed to the frame buffer 2 (Step S2), and saving thesucceeding frame of the image data (Step S3).

[0024] The frame difference detector 4 calculates an optical flowbetween two consecutive frames from the saved data of the frame buffers2 and 3 to detect a change in the image between the two frames. Thechange in the image is transferred to the barycenter calculator 5 whereit is used to calculate the barycenter of the change which is then savedin the barycenter coordinates memory 6. In the other barycentercoordinates memory 7, the barycenter is initialized to (0, 0).

[0025] The camera movement calculator 8 calculates a difference betweentwo outputs of the barycenter coordinates memories 6 and 7 to determinea horizontal pixel level shift ΔPx and a vertical pixel level shift ΔPy(Step S4). Then, the saved data of the barycenter coordinates memory 6is transferred to the barycenter coordinates memory 7.

[0026] This is followed by the horizontal angle of rotation controller 9and the vertical angle of rotation controller 10 calculating an angle ofrotation along the horizontal direction θx and an angle of rotationalong the vertical direction θy respectively from the resultant shiftsof the camera movement calculator 8 to control the point of view of thevirtual camera 11 (Step S5).

[0027] The virtual camera 11 acquires object data in the 3-D model spacefrom the 3-D object space database 12 and projects 3-D objects on thetwo dimensional screen as seen in its view. A resultant 3-D image isdisplayed on the liquid crystal display screen of the image display unit13 (Step S6).

[0028] The angle of rotation along the horizontal direction θx and theangle of rotation along the vertical direction θy for controlling thevirtual camera 11 at Step S5 may be calculated from the followingequations. It is assumed herein that the CCD camera of the image inputunit 1 is spaced substantially equally from subjects around by adistance d. The number of pixels along the horizontal direction in oneframe is Px and along the vertical direction is Py. The value along thevertical direction may be approximated from that along the horizontaldirection and the following explanation is based on the horizontaldirection. When the CCD camera of the image input unit 1 is turnedthrough the angle of rotation along the horizontal direction θx, theactual distance of movement on the screen is Lx.

θx=tan⁻¹(Lx/d)  (1)

[0029] As the shift of the pixel level in one frame is ΔPx in responseto the movement through θx, LX_(MAX):Lx=Px:ΔPx is established whereLX_(MAX) is an actual distance of movement. Then,

Lx(Δpx/Px)·Lx _(MAX)  (2)

[0030] When the equation (1) is then expressed by the equation (2),

θx=tan⁻¹(Lx _(MAX)/(Px·d))·ΔPx  (3)

[0031] There is now no variable. Similarly, along the vertical directionis expressed by:

θy=tan⁻¹(Ly _(MAX)/(Py·d))·ΔPy  (4)

[0032] Accordingly, if Lx_(MAX)=d=80 cm and the number of pixels alongthe horizontal direction Px is 160 at ΔPx=160, the camera is turned by45 degrees to the right.

[0033] The image displaying apparatus having the foregoing arrangementallows the horizontal angle-of-rotation controller 9 and the verticalangle-of-rotation controller 10 of the microcomputer 14 to detect amovement of the image display unit 13 through processing the input imagedata supplied from the image input unit 1 via the two frame buffers 2and 3 so that they function as a controller unit for remote-control ofthe camera view, and permits the virtual camera 11 and the 3-D objectspace database 12 to function in a combination as an image datagenerator for generating image data of the camera view remote controlledby the above mentioned controller unit. In operation, the image data ofthe camera view remote controlled in response to the motion of the imagedisplay unit 13 can be generated by the virtual camera 11 and displayedon the image display unit 13.

[0034]FIG. 3 is a front view of a mobile data assistant 100 associatedwith the present invention. FIG. 4 is a side view of the mobile dataassistant 100.

[0035] The mobile data assistant 100 is a hand-held computer including amicrocomputer 14 of the image displaying apparatus shown in FIG. 1. Themobile data assistant 100 has a small-sized video camera connected by avideo cable 17 to a video capture board 16 and operated as the imageinput unit 1. Image data produced by the image input unit 1 imaging thebackground of the image display unit 13 equipped with a liquid crystaldisplay is fed to the microcomputer 14 as a data input for detecting amotion of the image display unit 13. Similar to the operation of theimage displaying apparatus shown in FIG. 1, the mobile data assistant100 permits the image data of the camera view remote controlled inresponse to the motion of the image display unit 13 can be generated bythe virtual camera 11 and displayed on the image display unit 13.

[0036] In he mobile data assistant 100, the image input unit 1 for imageprocessing is separated from a controller of the virtual camera 11 toproduce and supply only data of the angle of rotation along thehorizontal and vertical directions for the camera 11 thus contributingto the versatility of the entire arrangement. This permits anycombination of three different units: the image input unit 1 for imageprocessing, the image display unit 13, and the camera 11 to becontrolled, for desired application.

[0037] The mobile data assistant 100 detects upward, downward, leftward,and rightward movements of a body of its image display unit 13 fromimage processing and uses their data to shift the view of the virtualcamera in a 3-D model space displayed on the screen of the image displayunit 13. Accordingly, the mobile data assistant 100 permits upward,downward, leftward, and rightward views in the 3-D model space to becreated as closer as seen by the human eyes while its mobility andoperability are improved.

[0038] The present invention is not limited to the prescribedembodiments where all the 3-D model space data is recorded in the 3-Dobject space database 12 provided as a fundamental function element ofthe microcomputer 14 in the main body. Any type of the 3-D model spacedata can be acquired from external sources. For example, themicrocomputer 14 may be communicated with the Internet through a modemport of a portable telephone not shown to receive a desired type of the3-D model space data from the Internet.

[0039] Also, the present invention is not limited to such a model asdescribed previously where the virtual camera 11 is virtually located ina 3-D model space for implementing the image data produced by an imagedata generator of the microcomputer 4 or a combination of the camera 11and the 3-D object space database 12 and displayed on the image displayunit 13. The image data generator may include an actual video camera,not shown, which is located in a real space and can be remote controlledby the controller for having a desired view.

[0040] Although the small-sized video camera of the image input unit 1in the mobile data assistant 100 is connected via the video captureboard 16 to the hand-held computer, it can be built integral with thehand-held computer.

[0041] As set forth above, according to the present invention, themovement of the image display unit is detected through processing inputimage data produced by the image input unit picturing the background ofthe image display unit and used for generating and displaying the viewof a camera remote controlled on the image display unit. This permitsthe image display unit to be always kept in the front of the eyes of auser through shifting the apparatus in any of upward, downward,leftward, and rightward directions.

What is claimed is:
 1. An image displaying apparatus comprising: animage display unit; an image input unit for picturing the image displayunit; a controller for processing input image data produced by the imageinput unit to detect a movement of the image display unit and for remotecontrolling the view of a camera in accordance with the detectedmovement; and an image data generator for generating data indicative ofthe view of the camera remote controlled by the controller, in which aresultant image generated by the image data generator can be displayedon the image display unit.
 2. An image displaying apparatus according toclaim 1 , wherein the image data generator is a virtual camera devicewhich generates the data indicative of the view of the camera remotecontrolled by the controller from object data in a 3-D model space whichis saved in and supplied from a 3-D object space database.
 3. An imagedisplaying apparatus according to claim 1 , wherein the image datagenerator is a microcomputer provided with a communication function sothat the data indicative of the view of the camera remote controlled bythe controller is generated from object data in a 3-D model space whichis obtained by the communication function.
 4. An image displayingapparatus according to claim 1 , wherein the image data generator is acamera device which is located in a real space and can be remotecontrolled by the controller for having a desired view.
 5. An imagedisplaying apparatus according to claim 1 , wherein the controllerdetects the movement of the image display unit from a change in thebarycenter of the input image between two consecutive frames produced bythe image input unit.
 6. An image displaying method comprising the stepsof: picturing the background of an image display unit; processingpictured data to detect a movement of the image display unit; generatingimage data of the view of a camera remote controlled in response to thedetected movement of the image input unit; and displaying on the imagedisplay unit the image data of the view of the camera controlled inresponse to the movement of the image input unit.
 7. An image displayingmethod according to claim 6 , wherein the image data of the view of thecamera remote controlled in response to the detected movement of theimage input unit is generated as the view of a virtual camera fromobject data in a 3-D model space which is saved in and supplied from a3-D object space database.
 8. An image displaying method according toclaim 6 , wherein the image data of the view of the camera remotecontrolled in response to the detected movement of the image input unitis generated as the view of a virtual camera from object data in a 3-Dmodel space which is obtained by the communication function of amicrocomputer.
 9. An image displaying method according to claim 6 ,wherein the image data of the view of the camera remote controlled inresponse to the detected movement of the image input unit is generatedby a camera device which is located in a real space and can be remotecontrolled.
 10. An image displaying method according to claim 6 ,wherein the detection of the movement of the image display unit includescalculating a change in the barycenter of the input image between twoconsecutive frames produced by the image input unit.